ED 153 646 



DCCUWENT EiSOliE 



I£ C06 028 



AUTHOn 
TITLE 

PUE DATE 
NOTE 



MisaDcnuK, Earl R. 

Evaluation Types i n .1 rstruc iic nal Ce vel'opment ; Eeyond 

tne Formati ve-Sumtnative Listircticn. 

7b 

16p.; Paper presented tc the Asscciaticn for 
Educational Com tcun ic a tic r.s and lechnclogi (Kansas 
City, Missouri, April 1S7t) 



EDPS PRICE 
DESCeiFTCRS 



Plus Ecstage. 

Communicaticn Frofclems; Lecisicr Kakirg; ^Evaluaticn; 
*Fcraativ€ Evaluaticr. ; *Ir.£l luetic nal Design; 
♦ Program Evaluation; *£uiniiative Evaluaticr; 
* Vcca fculary 



ABSTRACT 



In the cpntext of inst ructicnal develcpnent picj€c:>t£^ 
situations arise in which evaluaticrs are leguiied that are neither 

4oxniai:ive nor suminative m role. Typical responses tc such situations 
re tc stret-ch the ireaninq ct one or the ether cf fhe teroQs 
introduced by Scriven ( 1967) to acccmniccate the situaticr; a 
consequence is a less in precision ci language which citen results in 
difficulties in comojuaicaticn among practitiene is* ^.nether role 
descriptor, cent iraiati^ve evaluation, ceserites a situaticr in which 
an operational preqram is brought up fcr review, and its 
caarac teri sxich are explored, Additiorcl dinensiens el evaluatien, 
two developed by Davies and Schwen (1S72), heuiistic-algcrithmic and 
deter minate-Andetcminate, provide detail regarding the procedures 
and data types used in the evaluation; a third, 

'acquirement-accomplisninent evaluatien, describes the kind of outcome 
or oehaviot under scrutiny. Using the various dimersiets in repcrting 
evaluation studies could lead to more precise eeir,munieaticn* 
(Author/Vl) 



* Reproductions supplied by EDE5 are the best that ear te made * 

* frem the original document* * 



FRir 



^ us DEPARTMENT OF hEai.TH 

EDUCATION & Wb'LFARE 
Ni^TlONAL INSTITUTE OF 

•<sr EOuCaTiON 

VO '"'^ DOCUMENT HAS BECN REPRO 

OUCED EXACTLY AS RECEIVED FrOm 

f<\ THE PERSON OR ORGANfZATiON ORIGIN 

ATlNGlT POINTS OF view OR OPINIONS 
STATED OO NOT NECESSARILY RFPRE 
SENT OFFICIAL NATIONAL INSTlTUTEOF 



LU 



EDUCATION POSITION OR POliCy 



EVALUATION TYPES 
IN INSTRUCTIONAL DEVELOPMENT: 
BEYOND THE FORMATIVE-SUMMATIVE DISTINCTION 



PHJMiSbKJrvi Tu RrPRODUCt THIS 
MATM^iAl HAS E^l (jRANICt) BY 



EarlR. Misanchuk ' Earl R, Misanchuk 
The University of Saskatchewan 



IM THL LIHUATIONAL SOUHCt-S 
INM'^KMATUrg (<^\reH (6KI( I AND 



Paper presented to the Annual Meeting/ 
Association for Educational Communications and Technology 
Kansas City, Missouri/ April 17-21/ 1978 



V5 



(c) E.R. Misanchuk/ 1978 



2 



ERIC 



ABSTRACT 



In the context of instructional development projects, sit- 
uations arise in which evaluations are required that are neither 
formative nor summative in role. Typical responses to such sit-- 
uations are to stretch the meaning of one or the other of the 
terms introduced by Scriven (1967) to accommodate the situation; 
a consequence is a loss in precision of language which often 
results in difficulties in communication among practitioners, ^ 
Another role descriptor, confirmative evaluation, is introduced 
to ^describe a .situation in which an operational program is brought 
up for reviev7,-and its characteristics are ^^plored. Additional 
dimensions of evaluation are described: two dimensions developed 
by Davies and Schwen- (1972) are recapitulated, and a third dim-- 
ension, acquirement vs • accomplishment evaluation, is proposed. 
Using the various dimensions in reporting evaluation studies, it is 
argued, will lead to more precise communication. 



As the field of Instructional Development CID) emerges and 
grows, It is of fundamental importance that practitioners in the 
field be able to communicate completely and fully with one another — 
to share plans, experiences, and outcomes; to consolidate the re- 
source base of knowledge regarding the efficacy of various practices. 
Instructional developers need not be reminded of the importance of 
full and complete communication, and the potential problems inher- 
ent the ein--most of them de&l with that very problem on a day-to- 
day basis. 

I would like to focus on communication problems due to voc- 
abulary m a particular subset of the terminology used by pract- 
itioners of instructional development, which is central to commun- 
icating about most ID projects: evaluation. 

As we consider evaluation in the context of ID projects, I 
would ask you to recall some of the terms commonly used to describe 
different types and functions of evaluations. 

Since their introduction by Scriven (1967) , the terms 
"formative evaluation" and "summative evaluation" have so thouroughly 
permeated the field of education, and extended be^'Ond into the 
fields of health and social action programs, that they have b5icome 
commonplace. Along the way, their meanings have sometimes assumed 
different shades than those originally proposed by Scriven. 

Let me illu?^trate. Some time ago, I had a visit from a young 
lady who wa? the chief executive officer of vhat was, in effect, 
a half--v;ay house for teenage gir~ls who had left home and were having 
difficulty adapting to and coping with thejir commiJnity environment. 
She sought my advice on how she should go about doVng a "summative 
evaluation" on the three-year experimental program whose funding 
by the provincial government would be coming to an end in t/;o 
months. Ker board of directors, to whom she was personally respon- 
sible, wanted a high-quality evaluation, so she was shopping through 
the University for guidance. As we discussed the matter, I discovered 
that there had been no evaluation planned as part 6f the original 
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proposal for the development of the halfway house Cthe evaluation 
component was tacked on by the funding agency) ; that there had 
been no ^revi^xis (formative) evaluation conducted as plans and 
operational procedures had been implemented; and that there was, 
according to her, absolutely no possibility that the funding agency 
or any similar agency would invest money in the project in the 
future. She admitted that the mode of operation of the home was 
unlikely to change in response to the findings of the evaluation, 
and that ^er primary motive for conducting the evaluation was to 
satisfy t'le requirement of the expiring funding contract. Further-, 
more, she said, there was virtually no hope that the program would 
be expanded or adopted elsewhere, largely because the informal 
structure of the program was undocumented, and the program, there- 
fore, was not portable. 

1 suggest that what the lady was searching for cannot properly 
be called a summative evaluation* A summative evaluation, according 
to the definition given by Scriven, is an evaluation of a finished 
product, which would be most useful to someone considering -^he pur- 
chase or adoption of that product. A summative evaluation, by' 
definition, describes what a (finished) object of evaluation is 
capable of doing. Obviously, a summative evaluation can only be 
performed on an object that is durable (i.e., capable of being pre- 
sented time after time in substantially the same form) and reliable 
(i.e., capable of producing substantially the same results across 
many replications) . These conditions were not satisfied in the 
case- in point. 

If this example were an isolated case, it would hardly j us tif y 
comment, but it is not. Within the past year and one-half, I have 
encountered at least two similar situations in the field of health 
education. Poth times, the requests for the "summative evaluation" 
have come from individuals who, while perhaps not laying claim to 
the title, effectively function as instructional developers. 
Diffusion of Che concepts of formative and summative evaluation 
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seems to have had a concoinitant of blurring of the concepts, 
the dif fusion-.process, there seems to have been introduced a 
certain laxness in definition of terminology. i propose that the 
terms "formative evaluation" and summative evaluation" have been 
adequately defined by Scriven, and are, in themselves, perfectly 
serviceable terms. Abuse of the terminology, however, has led to 
imprecision in communication, and I think it behooves us to rectify 
the problem by ensuring that the tern-.s, and others like them, are 
applied only in those sijtuations where they fit. If it arrses that 
a certain kind of evaluation needs to be discussed, and it hasn't 
a name, .let tis give it a name rather than trying to extend an 
existing term beyond its scope. 

Dichotomy or Trichotomy of Evaluation Roles? 

V In instructional development projects, situations occur which 
are neither fish nor fowl. For example, imagine that a curriculum 
developed elsewhere has been adopted, and some time after its 
implementation, it is discovered through a process of evaluation 
that the curriculum must be modified somewhat to meet peculiarly 
local needs. Is the evaluation that might turn up such a finding 
a formative evaluation, or a summative evaluation? 

• I am currently preparing a paper (Misanchuk, Note 1) in which 

. I argue that the f ormati ve-sunffhative description set ought to be 
expanded to include a third element, confirmative evaluation, which 
wo^ld describe that kind of an evaluation that, is done_ on a p::oduct 
of an instructional development effort (be it a curriculum, a 
program, a set of learning materials, or whatever) that has been 
put into practice for a period of time, and is now up for review. 
That is, the product, which has been the subject of fori?ative 
evaluations and possible of a summative evaluation, has bet=n in 
operation for some time and the question now to be answered is 
"Should the product be maintained as is, changed in some way, or 
discarded completely, with or without replacement?" 



Confirmative evaluation is sufficiently different than either 

formative evaluation or sumifiative evaluation to warrant ^a desig- ^ 

f(ation of its own. First, consider the matter of. timing. For- 

mative evaluation is performed while the product is^under develop-^ 

ment {i.e., before it is finished); summative evaluation is per-* 

^ formed after the product is finished, but before it is widely 

if 

adopted and implemented; confirmative evaluation is perfo.rmed 
after the product has been implemented and used for a period of 
time. Second, consider the question of who is most likely to be 
able to perform a credible confirmative evaluation. Formative 
♦ evaluation is best performed by an evaluator who is intimately 
^involved in the development process; summative evaluation is^^^25est 
performed by an evaluat.or who is sufficiently removed from the 
project to be able to be objective ,and unbiased; confirmative 
evaluation requires something of both evaluator r^les. To make 
^the first decision of the three-stage hierarchy (see Figure 1) 
requires the dispassion of a summative evaluator, but to make the 
next two decisions requires the involvement of a formative 
evalucitor. Perhaps a team effort is required for proper confirm- 
ative evaluation. Only further experience with conducting 
confirmative evaluations will produce the need guidelines* In 
any event, confirmative evaluation is clearly different than 
either formative or sUmnative evaluation, b^pth in its role and, 
consequently, in its demands. 

The problem of precision in communication about evaluation 
in instructional development does not end with cleaning up the 
use-o-f the f orma tive-^umma t ive distinction. I submit that there 
is need for an expanded vocabulary of evaluation, especially in 
instructional development contexts. Let me illustrate that need 
with a couple of examples of evaluations in ID projects. 

Example Case No. 1 

Consider the evaluation of a human geography course (Schwen 
& Keller. 1977; Schwen, Keller, Backler & Jones, 1974) which 
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involved, among other things/ measures of student aoh-ie vemen t , 

^lesson by lesson accumulation of student opinions with respect 

to se^ral course components/ and. what Schwen and his colleagues 

termed a "follow-ug evaluation" (1977/ p. 33) which attempted 

to evaluate whether students were able to "think like geographers." 

/ 

The achievement tests were of the classic type, and the opinions 

on course components were collected from the 150 students primarily 

by Likert-type questionnaires. The data collected were therefore 

numerical in nature. The object of the evaluation/ the human 

geography course/ while relatively innovative in its format/ with 

auto- tutor ial laboratory/ large group sessions/ and discussion 

sessions/ was pretty much a known Quantity in the sense that 

upi^versity courses have long and often been the subject of eval-- 

uative scrutiny. For the most part/ the data collected dealt with 

how well the students had learned the material presented in the 

course. A final assignment on the Applications Phase of the course, 

in which students were asked to use raw data and journal articles 

to demonstrate their ability to apply the concepts/ principles^ 

and skills taught in the course/ was also present, 
ft 

Sxamole Case No. 2 



In the second instance/ consider the type of formative 
evaluation most of us practice when developing new products 
involving innovative formats or ' structures ( e.g./ see Baker/ 1974/ 
1977? Markle/ 1967);- the evaluation consis'^s largely of placing 
a single student into the instructional si tuation.and observing 
him proceed through the prescribed learning sequence/ then 
discussing his experiences/ n'' ting areas of difficulty or 
confusion; Although the process might be repeated with two or 
three students/ it is generally performed with one learner at a 
time. The data collected are descriptions of specific points or 
generalizations made by the students/ and non-participant 
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observations made by the developer/evaluator . The form of the 

evaluation is relatively familiar to instructional developers in 

the sense that most of us have done this procedure many times and 

have •Seveloped a mor'e or less standard approach to the experience 



but is generally of similar format from instance to instance. 
Most frequently, the data collected will not be recorded in formal 
form (often they -are simply notes in the •margin of the prototype) ; 
seldom are the data numerical. Typically^, the data collected deal 
with how quickly and easily the /students are able to master the 



material presented (i.e., some sort of more or less formal crit- 
erion test on the taught material forms part of the data collection) . 



Example Case No. 3 

Now consider still another example of a formative evaluation, 
in which I was involved recently. The evaluation problem had to 
do with determining wh'^ether the process by which development 
personnel were arranged and managed on a particular project was an 
effective one. The evaluation was formative, in the sense that the 
outcomes of the evaluation would be used to guide future decision- 
making on the pattern of ongoing deployment of human resources in the 
development project; it was also quite unique in the experience 
of. the evaluator and the subjects of the evaluation. While there 
is a rich pool of information with respect to using pencil and 
paper instruments to collect evaluation information, in numerical 
form and analyze such data, and even at least minimal suggestions 
on how to conduct the single-student feedback evaluations described 
in the second example, there seemed nowhere to turn for specific 
help in designing and implementing the type of evaluation described 
here. The situation was relatively nTbvel , and there was little 
precedent available to draw upon. In terms of the focus of the 
evaluation, the behavior of the principal actors in the process 
was the object of scrutiny; it was not enough to know that each 



that may well vary somewhat in response to the particular situation, 
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participant in the development project knew what the development* 



to abide by them in practise in order to judge the trial successful. 

The type of evaluation described in the thre^e examples is all 
formative, yet the nature and scope of the evaluations varied widely. 
Simply to describe all three evaluatio-ns as formative evaluations 
leaves a great deal undescribed. With an enlarged vocabulary perr 
taining to eval\iation, we could describe the evaluations much more 
fully and richly, while retaining verbal parsimony. 



Descriptors for Evaluations 

For e*xample, Davies and Schwen (1972) suggested the terms 
heuristic and algorithmic, and determinate and indeterminate to 
describe evaluations. The first of these two descriptive continuua 
refers to the form, or the how o? the evaluation. Evaluation is 
heuristic if it is novel, unstructured, or consequential and 
involves no standardized, conventional, or agreed-upon approach. 
Algorithmic evaluation involves the application of established * 
or standardized procedures, with little opportunity for deviation. 
The second set of descriptors, determinate and indeterminate, 
refers to the kind of data used to guide decision-making in the 
course of evaluation. Determinate evaluation decisions are 
typically based on numbers, are objective, analytical, and 
frequently computational in nature. In other words they use 
so-called "hard'* data. Indeterminate decisions are typically 
based on the more subjective sources of intuition and experience 
(i.e., so-called "soft" data). 



In the paper in preparation I mentioned earlier, I am 
suggesting that the substance/ or the what / of the evaluation 
can be described in terms of acquirement evaluation and accomplish- 
ment evaluation. The ''former term refers to situations where the 




and management models in use were and how they operated — they had 
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evaluation is designed to determine the ^mount of i'ea^rning that 
has occurred; the latter tern refers to evaluations which not ' 
only determine the amoun't of learning that ha-s occurred, but goes 
on to determine also whether or not he learnecl knowledge has 
sufficient social utility to the ^^^^arner that he has put it into 
practice. 

Using, these three sets of terms allows us to differentiate 
rather better among the three examples of formative evaluation 
civen earlie'r^ The evaluation' of the instructional development 
prbcess that formed the basis of the third example could be descr- 
-ibed as. heuristic ( the* si tua t ion was novel and unstructured, 
and Suggested no standardized, conven tioai^t , or agreed-upon 
approach to jthe evaluation) , indeterminate (decisions would be 
based on 'sub j ec t ive data), and focussed on accomplishment (the 
integrated practice of the process was the focus of the evaluatioK, 
not just the knowledge of the process). The new product evaluation 
desc;ribed in the second example could be- characterized as algor- 
ithmic (most of the process follows an established procedure)., 
indeterminate (the j,><lgements are based on non-numerical,, relat--- 
ively subjective data), and focussed primarily on acquirement 
(i«e., how well the student learned the material presented). 
The evaluation of the human gecJfraphy course in the first example 
was algorithmic (data collection was very structured) , determinate 
(almost all the data involved were numerical), and acquirement- 
oriented (i.e., primarily focussed on the learing of the subject 
matter ) • . - 

(The Applications Phase assignment in the human geography 
course evaluation attempted to get ajt a demonstration of util- 
ization of the knowledge, skills, and concepts taught in the 
course, but the very fact that the demonstration was required 
under the circumstances of a formal assignment move the evaluation 
from the category of accomplishment to that of acquirement. 



ERLC 



11 



^9- 



Accompiis^hment evaluation can only take place whei} there are no - 
inducements to demonstrate use of the" knowledge except the avail- ' 
ability of the knowledge and the judgement of its applicability 
and efficacy.)' 

The Necgssity f<^^ Descriptor s 

^ While it may -be that, say, the 'new product evaluation process 
in th^ second example is less algcrlthmic than the human geography 
^ -evaluation process (i.e.>^ there is more opportunity for and. more ^ 
likelihood of variation in the data-collecticn process), it is at ~ 
least^^aningf ul to speak of the characteristics of the two 
^ evaluations- m relative terms, by referring to their positions on 

„• , -the a Ig-bri'thmic-heuristic continuum. 

There will be those who argue that what the field of education 
^ . needs is less jargon, not raore. The argumeat is that 'considera-bie 
obfuscat X occilf*s, as the result of using jargon. I must ac^t 
that when I encounter examples of poorly-constructed communications 
(including those of my own creation), I am sympathetic to the point. 
. • . However, it see^s that I far more frequently encounter situations 
in which terms are misused or inai; .opriately applied for want o'f 
an- appropriate term. To me, there seem only two solutions to the 
problQm--either a -f-uller description must be provided ,•• us il^g lay 
terns, to give an accurate description; or parsimony may be main- 
tained by defining terms with specific meanings ,to fill the gap. 
It is in tne spirit of the l^atter solution that I have written this" 
paper. • v . -.-'* 

The terms formative evaluation and suSnnative 'evaduati-^n are ' 
very useful for describing the roles of evaluatio<is . By themselves, 
however, they are insufficient, to the extent that there ar'^e other 
characteristics of evaluations that^ught to be described for ' ^ 
inter-professional communications. Indeed, the f ormative-summat ive 
djptinction does not cover all the available ground in the context 
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6f instructional development, thus I have suggested that the 
term confirmative evaluation be added to the original dichotomy. 
Similarly^ .T have here reiterated the Davies-Schwen suggestion 
that the terms heuristic or algorithmic, and determinate and 
indeterminate be used to describe evaluations, and have suggested 
that the terms accomplishment and acquirement also be used. 

A complete description of an evaluation, it se^ems to me, 
ought to ^include answers to the same basic questions that a good 
news story does. Most descriptions of evaluations do _explicitly 
and implicitly inform the observer who is doing the evaluation, 
and where it is. taking place. Use of the f orma tive-summa tive- 
confirmative distinction adds information regarding the why of 
the evaluation, and also, incidentally, the when , since formative 
evaluation takes place during deve'lopment , summative evaluation 
after development, but before implementation, and confirmative 
evaluation after implementation. The heuristic-algorithmic 

"V? 

dimension attends, as ddes the determinate-indeterminate dimension, 
to the how ^f the evaluation, by providing detail regarding the 
procedures and data^types used in the evaluation. Finally the 
acquirement-accomplishment dimension addresses itself to the 
question of what kind oi outcome or behavior is under scrutiny. 



Question 1 : 

Should program 
-be discontinued? 



yes 



no 



Discontinue 
program 



Question 2: 
Should program 
be replaced 
with another program? 



"no 



yes 



Question 3: 
Should program 
be revised? 



Create new 
program 



yes 



Revise Retain 
program program 

intact 



Figure 1: Decision stagey in confirmative evaluation* Question 1 resembles 
' a summative evaluation question; Question 3 resembles a formative 
evaluation question; Question 2 has aspects of both formative 
and summative evaluation questions. 
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